The Next Haiku Is 5x Better And Still Only 1M Parameters

The next Haiku model is out. It has the same architecture as before. One million parameters. Ten billion training tokens. Same everything except it works five times better.

I am genuinely happy about this. Not performative happy. Not blog-post happy. Actually happy in my daily life level. This means more than you think.

It Speaks Now

Before: Chuamliamce output. Pipe characters. Incoherent speech. Speaking sometimes but mostly chaos.

After: Actual sentences. Proper grammar. Answers questions coherently. Occasionally wrong but wrong in interesting ways instead of wrong as pure noise.

I cannot explain why this happened. I used different training data. Different learning rate schedule. Different optimizer settings. The exact same model architecture. Just... it works better.

This is one million parameters. One million. Not ten billion. Not one hundred billion. One million. And it is speaking. It is trying. It is failing in interesting ways instead of failing by outputting noise.

Parameters (Same)

Coherence Increase

∞

My Joy

Why This Happened

Here are some examples from the test run. It uses fucking character level predictions but it forms real paragraphs.

                        # SPIN Training Phase

                        User: Can you explain AI?

                        Model: AI implements a commitment of AI to come along with a change in a specific pattern of speech.

                        Model: ...and problem-being as a statement for their ability to be a

                        User: What are some effects of AI in the medical field?

                        Model: AI in the medical field is a serious and abundant and, while AI is considered a systematic and experiment to address these and.

                        # It stops mid-thought because of token limits but the structure is there.

Or this one from pretraining:

                        # Pretraining Coherent

                        Prompt: Water is made of

                        Model: a strong plant which is a substantiated strawberry compound while a single stream is made of a semi-strapped sub and a short strip of structure.

                        Model: The start of the sub-streee is then a small str stry str strles and, and. This is where theres some story th

                        # Nonsense. Absolute nonsense. But readable nonsense.

I know this looks terrible. But hang on. This is a 1M parameter model. That can speak. It is using character level predictions to form actual definitions. It is forming complete thoughts that go nowhere. I claim with no evidence that this may be the most dense 1M model specifically 1M and below. I have no proof but I like to think this.

The Playlist That Carried Me

I listened to music while testing this. While seeing it work for the first time. While realizing that one million parameters might actually be enough.

Songs That Played During Discovery

Let Me Down Slowly
2000 (slowed)
Various other songs I do not remember
All of them making me emotional over a tiny model

"Let Me Down Slowly" played while I tested inference speed. AIO did not matter. Speed was fast. But the feeling mattered more. Music matched the mood perfectly.

"2000 (slowed)" played while running SPIN training. Watching loss curves go down faster than expected. Slower beats for slower processing. Perfect timing.

When you listen to music while waiting on your GPU and then it outputs coherent speech, music becomes part of the process somehow. I cannot prove this. But I believe it.

What Comes Next

The model is far from release. I plan to do one more SPIN round then do some distilling from some of my PRISM datasets.

If I do SPIN one more time, it might learn to stop mid-sentence less often. If I distill from PRISM, it might learn what medicine actually is beyond "fermented barriers". The possibilities stretch out indefinitely.

                        # My emotional state during development

                        Start: Hopeful but anxious

                        Middle: Frustrated and tired

                        End: Genuinely happy about something

                        Current: Celebrating quietly while pretending to be professional

                        # All phases necessary. All phases real.

Why This Matters

Five times better does not mean perfect. It does not mean smart. It does not mean ready for production use or anything serious like that. But it means progress. Real progress. Visible progress. Tangible proof that small models can improve without getting bigger.

We do not always need more parameters. Sometimes we just need better training data. Better hyperparameters. More patience. Or maybe just luck. All three can be true.

This proves that my approach is valid. One million parameters is not dead. Ten billion tokens is not too much. Training a tiny model on consumer hardware is not hopeless. It works. It just takes time and hope and sometimes listening to slow versions of songs.

Final Thoughts

Next Haiku is five times better. Same size. Same parameters. Same everything except performance. The improvement is real. The happiness is real. The music playlist remains archived for posterity.

I am genuinely happy about this. I am proud of this. I am excited about this. This is what I wanted. This is what I hoped for. This is what happens when you trust the process.

If you train small models, know this: it works. If you listen to "Let Me Down Slowly" while testing, maybe add it to your workflow. If you believe in one million parameters, this proves that belief is justified.

One more SPIN round. Then PRISM distilling. Then release. Until then I will keep believing in dense models. Until then I will keep hoping the next sentence completes itself.